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ABSTRACT 

The objective of the project was to pilot test the 
Model to Evaluate Inservice Personnel Development. The pilot test was 
designed to determine: (1) whether the model could evaluate the 
personnel development programs in the two Stites where the model was 
pilot tested, (2) applicability of the model in other States, and (3) 
the potential of the model, with modification, to evaluate projects 
other than the kind funded under Section 553 of Part F of the 1968 
vocational amendments. The pilot test was conducted in two phases. 
The first phase was a test of the components and procedures of the 
model in Tennessee with a list of the changes resulting from that 
phase included in the report. Phase 2 was conducted in California, 
and additional changes resulted from the pilot test there. It was 
concluded that more time was needed to conduct the pilot test, but 
using two different States and numerous reviewers was beneficial. The 
final part of the report states four recommendations for the 
dissemination of the evaluation model. The remaining one-half of the 
document consists of an appendix of observation and interview 
schedules. (BP) 
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This document is the final report of the project entitled "A Pilot 
Test of an Evaluation System for Vocational Education Leadership and 
Professional Development Activities." The text of the report is arranged 
in the format for final reports contained in Appendix A of the Handbook 
for Director s of projects funded under parts D and F of the Education 
Professions "evelopment Act (U. S. Office of Education, January 15, 1972). 
The evaluation model tested and subsequently revised is submitted with 
this report 

1. Restat e all of the objectives (anticipated outcomes) as originally 
sta^t-1 in the plan of operation or, if applicable, as modified 
during the operation of the project . 

The objective of the project was to pilot test, in Tennessee 
and California, the Model to Evaluate Inservice Personnel Develqp- 
ment in Vocational -Technical Education. The evaluation model was 
developed in the U. S. Office of Education-funded project entitled 
"An Evaluation System for Vocational Education Leadership and Pro- 
fessional Development Activities." 

The pilot test was limited because of the short amount of time 
and limited funds available for it. Although most of the compon- 
ents and procedures were tested, two were not. Phe data collected 
with the survey forms were prepared by the project staff, not by 
state personnel as was proposed in the model. Instructions for the 
preparation of data, both by hand and by computer, were written 
after the pilot test. Also, the state evaluation team consisted 
of three rather than five persons, and team members conducted the 
meeting in the presence of project staff. Although project staff 
members were only observers and did not participate in the meet- 
ing, questions of clarification were sometimes asked of them. 

The anticipated outcome of the project was a tested model which 
could be used by most, if not all, state divisions of vocational 
education to evaluate personnel development activities. More 
specifically, the pilot test was designed to determine: (1) 
whether the model could evaluate the personnel development programs 
in the two states where the model was pilot tested, (2) applica- 
bility of the model in other states, and (3) the potential of the 
model, with modficiation, to evaluate projects other than the 
kind funded under section 553 of part F of the I968 vocational 
amendments . 

2. State the actual outcomes of the project or sub -project and 
describe the instrument s used to measure the outcomes . 

The pilot test was conducted in two phases. The first phase 
was a test of the components in two phases. The first phase was 
a test of the coniponents and procedures of the model in Tennessee 
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during April and May of 1972. As a result of this phase, several 
changes were made in the evaluation model. A list of the changes 
resulting from that phase follows: 

a. The model was edited to clarify the meaning of both materials 
and procedures. For example, wording was changed to make 
mp*«»rials more readable. Also, instructions were given to 

- the state coordinator of personnel development send a 
project summary or abstract as advance material to members 
of the state evaluation team. Finally, a sampling informa- 
tion sheet was included with the data, to provide figures on 
population, sample size, and percentage responding for each 
project and for the entire program. 

b. The m.)del was changed to include provisions for state personnel 
to send advance information to members of the state evaluation 
team (SET) before its meeting. Accordingly, the staff sent 
descriptions of personnel development projects to be evaluated 
in California and procedures of the evaluation to members of 
the California SET (second phase of the test) before they met. 

c* The substantive focus of the evaluation model was modified 
somewhat. More emphasis was placed xxpon data related to the 
achievement of program and project objectives. In addition, 
data were no longer broken out according to the economic indi- 
cators of the agencies in \diich the project participants worked* 

The second phase of the pilot test was conducted in Cali- 
fornia during May and June* The following are changes in the 
model that resulted from both phases of the pilot test, but 
particularly from the California phase. 

(1) A preface was added to the model. 

(2) Instructions for making and using program and project 
description sheets were included a 

(3) Instructions for the SET meeting were modified. 
--Provisions for orientation of the SET members were in- 
cluded. 

--The role of the state coordinator of personnel devel- 
opment was described in more detail. 

--State evaluation team members were specifically instruc- 
ted to use their professional judgment in dealing with 
the evaluation data. 

--The state coordinator of personnel development was 
instructed to select one of the SET members as chairman 
of the team when he selected the team. 
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--The SET chairman was given the responsibility to chair 
the meeting and insure delivery of the evaluation re- 
port to the state director of vocational education. 

d. Sfc/eral changes were made in the instructions for distributing 
nnd collecting data, analyzing data, and conducting the SET 
meeting. The changes were made to clarify the meaning of 
instructions through better wording and the addition of infor- 
mation whic; had not been included before. 

e. The computer program was edited. 

f . The surveys of the model were edited to make them more consis- 
tent with the analyzed data presented to members of the team 
during the SET meeting. 

g. In several components of the model, instructions and descrip- 
tions emphasized that the model can be altered to fit the 
individual needs of states using it. 

Project staff members used interview and observation schedules 
to evaluate the operation of the evaluation model in the pilot test. 
The observation schedule is attached as Appendix 1 and the inter- 
view schedules are Appendix 2. Data from the schedules were used 
to make the modifications described above. 

The staff observed the state evaluation team meetings in both 
Tennessee and California. In addition, they observed the flow of 
surveys into and from the field in order to determine the clarity 
of instructions for collecting data with the survey forms of the 
evaluation model. These observations resulted in the detection of 
several minor errors in the instructions which made it difficult 
for either the state coordinator of personnel development or the 
SET members to perform certain necessary functions in the evalua- 
tion. 

The staff interviewed the state director of vocational educa- 
tion in Tennessee, the state coordinators of personnel development 
in Tennessee and California, and members of the state evaluation 
teams in both states. These persons provided support for numerous 
pr edures and materials of the model and made suggestions for 
changes in others. Persons from both states indicated it was very 
likely that they would use the evaluation model during the next 
fiscal year to evaluate personnel development in vocational educa- 
tion. 
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3. State the reason for discrepancies between the antic if.pated and 
actual outcomes . 

There were no discrepancies between anticipated and actual 
outcomes. Results of the obse??vatioa8 and the Interviews sug- 
^ gested that the model will function in many situatlcr.3. The pilot 
test did not, however, reveal the types of situations or circum- 
stances in which the model will not work. State coordinators of 
personnel development in the two states indicated that data ob- 
tained through the evaluation would be helpful to them in planning 
future programs. Also, the pilot test did not yield information 
on the potential of the model to evaluate projects other than the 
kind funded under section 553 of part P of the I968 amendments. 

h. If plans have beefl made for follow*up evaluation , indicate the 
date when additional information will be available . 

There are no plans for a follow-up evaluation at this time. 
The evaluation model is submitted with this report to the Office 
of Education pursuant to the terms of Grant Number OEG-0-72-0051 
(725), Project Number 02-2592. 

5 • State any observations which might be helpful to others consider - 
ing embarking on a venture such as tHis one • 

a. MDre time is needed to conduct a pilot test such as this. The 
late funding of the grant award did not allow sufficient time 
to contact states, negotiate agreements to conduct the pilot 
test of the total model. As a result, the California phase of 
the pilot test was not completed until two weeks before June 30, 
the original project deadline. This did not allow sufficient 
time to make the changes in the evaluation model necessary as 

a result of the California phase of the test. 

b. The use of two states with very different personnel develop- 
ment programs was helpful. In this case, the staff received 
indications of how well the evaluation model would work in a 
state with a large program and one with a small program. 

c. The use of numerous reviewers and critics to react to the model 
and parts of it was particularly beneficial in the development 
of the evaluation model. Some 20 persons of the 30 who were 
asked reacted to the materials. 

6 . State any final recommendations which WDuld be useful to the bureau 
jji administering the program under which you received your grant . 

The recommendations which follow do not relate to the adminis-* 
tration of the program but to the diseemination of the evaluation 
model. It is recommended that: 
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a. The U. S. Office of Education provide The Center for Vocational 
and Technical Education the funds to produce 200 sets of the 
evaluation model. 

b. The Center for Vocational and Technical Education distribute 
copies of the model oh a specimen basis in the folloving man- 
ner: 

—One copy to each of the state coordinators of personnel 
development in the ^0 states. 

--Additional copies to key teacher education departments sug- 
gested by the state coordinators • 

c. The Center communicate the limitations on validation of the 
model to users to insure that they do not believe extensive 
field testing has been conducted. 

d. The Center provide a brief orientation workshop, funded by the 
U. S. Office of Education^ for training state coordinators and 
others willing to pay travel and lodging expenses in the use of 
the model. 



ERIC 



APPENDICES 



/jpPENrn 1 

Ctoservation Schedule 



8 



Oldservatlon Schedule 
(S£T Meeting) 



1. There is little discussion designed to clarify the functions 
of the SET. 



agree 



disagree 



2. Use the space which follows to record the number of positive 
remarks made hy SET members about Information collected In the 
evaluation or elicited by the questions in AID . (Positive remarks 
are those which are favorable toward the iiifoznatlon . ) 



3. Use the space which follows to record the number of negative 
remarks made by SET members about information collected in the 
evaluation or elicited by questions in the aid for interpreting 
data (aid). (Negative remarks are those which are neither 
favorable nor neutral toward the information. ) 
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SET meiDbers draw conclusions about state programs easily. 



0 - no conclusions are made 

1 - tew (1-3) are reached slowly 

2 - few (1-3) €ure reached qidckly 

3 - some (^-6) are reached 

k - several (more than 6) are reached 



The conclusions are well-founded 



0 


1 


2 


3 1 ^ 











0 - no basis for conclusion 

1 - seem arbitrary 

2 - partially based upon data 

3 - most seem based on evidence 

k - all seem based on evidence and seem logical 
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6. The ^state evaluation team easily made recommendation which 
seemed to follow from the data considered in the meeting. 



0 - disagree 

1 - disagree slightly 

2 - neither agree nor disagree 

3 - agree s light ly 
k - agree 



7. What has happened as a result of the evaluation report? 



8. A group of judges will rate the evaluation report on the basis 
of criteria developed by the project staff. 
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APPENDIX 2 
Interview Schedules 
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Interview Sohedule 
(SET Menibers) 



1. Were instructions for the evalxuition clear and understandable? 
(probe) 



2. Was the aid for interpreting data (AID) useful to you? 



3. Rate AID on its usefulness on a scale of 0-U with 0 being of 
no use and h extremely useful. 
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U. How could the SET be helped to perform its functions? 



5. Did you have adequate tiitte to perform the SET functions? 



6. Is the model presented in a logical manner? 



7. How much training will be required for SET's to use this model? 
What kind of training should be given? What advance literature 
should be provided? 
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Inteirview Schedule 
(state Coordinator of Personnel Development) 



1. Are the sampling procedures worlcable? 



2. Can data be collected in the amount of time allowed in the 
pilot test? 



3. Will states conducting the evaluation have any particular 
problems in securing a high response rate (80^ or more)? 
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k. Are the procedures used to select the SET workable? 



3. Is the model organized logically? E.g. , the seven documents? 
format of documents? That part they use? 



6. Can the model be used In programs of different size with 
several different kinds of projects? 



7. Will the cost of this evaluation be reasonable? 
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y. Can the evaluation operate within the time frame of the state 
program to provide information for planning future programs? 
Vniere should the evaluation be conducted? 
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9. What training will be necessary for states to use the 
evaluation model? 



10. Can the procedures of the evaluation model be used by other 
states? 



11. What information provided by the SET was most useful? 
Least useful? 
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12. VJhat additional information should te included in the evaluation? 
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